Virtual machines (VMs) have widely been used in computer systems as one of computer virtualization technologies in recent years. In a virtual machine, the hardware (resource) of a computer is virtualized, and different programs may run on the virtualized hardware. Each virtual machine runs completely independently. Accordingly, use of virtual machines allows a single computer to function as multiple servers. Virtual machines are created by a virtual machine monitor running on the computer. The virtual machine monitor allocates hardware. Through the virtual machine monitor, each virtual machine performs input/output (I/O), that is, accesses the hardware. However, using the virtual machine monitor reduces the speed of access to the hardware and, in particular, the speed of access to an I/O device which is connected to the computer and which receives or outputs data from or to the computer. For this reason, to improve I/O performance, a technology called “direct I/O” that directly allocates I/O devices to virtual machines has recently been implemented as a hardware function. An I/O device used in direct I/O will be hereafter referred to as a “peripheral component interconnect (PCI) device.”
In direct I/O, direct DMA (direct memory access) is performed between a virtual machine and a PCI device using direct I/O hardware, which is hardware for performing DMA transfer address conversion (DMA Remapping) between the respective physical address spaces of the virtual machine and the computer. The PCI configuration space related to the setting of the PCI device, a device interrupt, and the like are emulated and reported to the virtual machine by the virtual machine monitor. As a result, the virtual machine may be handled as if an actual PCI device were directly connected thereto.
Virtualization technologies related to virtual machines include a technology called live migration that moves a virtual machine running on a computer to another computer. Live migration is used such as in order to distribute the load in operation management or cope with failures. In live migration, all states of the virtual machine, including the states of the CPU and the hardware register, the memory contents, and the state of I/O transfer, are required to be the same before and after migration.
Generally, live migration may not be performed on a virtual machine using a PCI device by direct I/O because it is common practice that the PCI device being used by the source computer is not connected to the destination computer.
Recently, it has been proposed to use an I/O switch which allows multiple computers to share the connection to a single PCI device (hereafter referred to as a “PCI switch”) and to perform live migration of a virtual machine using direct I/O between the computers connected to the PCI switch. “Share” as used herein does not mean that the computers can use the PCI device simultaneously. This means that switching the connection in the PCI switch allows changing a computer to be connected to the PCI device, without changing the physical wiring. With regard to the sharing of a PCI device using a PCI switch, a standard, multi-root I/O virtualization (MR-IOV), is laid down by the PCI-SIG.
The following three steps have been performed conventionally in order to perform live migration of a virtual machine using direct I/O between computers connected to a PCI switch as described above:
(1) Pre-Copy Phase
A non-operating virtual machine is created on the destination computer, and the memory contents of the virtual machine running on the source computer are transferred to the destination computer and then copied to the created virtual machine.
(2) Stop-and-Copy Phase
When the amount of the transferred memory contents exceeds a threshold, the virtual machine running on the source computer is suspended. During the suspension, the remaining memory contents, the memory contents changed after the transfer is started, the CPU state, the state of the emulated hardware, and the like are transferred to the destination computer. Then, in the PCI switch, the PCI device being used using direct I/O is disconnected from the source computer, and then connected to the destination computer, allowing the destination computer to access the PCI device using direct I/O.
(3) Activation Phase
The destination computer incorporates the contents transferred in the Stop-and-Copy phase into the created virtual machine and restarts the virtual machine. The source virtual machine is discarded.
The reason why the source computer and the PCI device are disconnected in the Stop-and-Copy phase is that the same PCI configuration space is not allocated to the multiple computers. Thus, the virtual machine to be migrated is suspended between the time when the source computer and the PCI device are disconnected and the time when the virtual machine is resumed in the destination computer. During this suspension, processes are performed such as hot-plug disconnection or connection of the PCI device from or to the source and destination computers, PCI device configuration by firmware such as BIOS after the disconnection and connection are reported by the PCI switch, and direct I/O setting by the virtual machine monitor. For this reason, while the virtual machine not using direct I/O is suspended for several tens of ms to about 200 ms during live migration, a virtual machine using direct I/O is suspended for another several hundred ms during live migration.
A longer suspension time may cause time-out or the like of a virtual machine used to provide services, reducing service quality. As described above, live migration is often performed such as in order to distribute the load in operation management or cope with failures. In this respect also, it is important to reduce the suspension time associated with live migration of a virtual machine using direct I/O.
Japanese Laid-open Patent Publication Nos. 2010-39685, 2009-181418, 2009-217608 and 2008-46722, International Publication No. WO 2010/116402, and “Intel (R) Virtualization Technology for Directed I/O Architecture Specification Rev 1.2,” http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf) are examples of related art.